{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Label Mistakes with FiftyOne\n",
    "\n",
    "Annotations mistakes create an artificial ceiling on the performance of your models. However, finding these mistakes by hand is at least as arduous as the original annotation work! Enter FiftyOne.\n",
    "\n",
    "This tutorial shows how FiftyOne can help you find and correct label mistakes in your datasets, enabling you to curate higher quality datasets and, ultimately, train better models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this walkthrough, we explore how FiftyOne can be used to help you find mistakes in your annotations.\n",
    "\n",
    "We'll cover the following concepts:\n",
    "\n",
    "-   Loading your existing dataset in FiftyOne\n",
    "-   Adding predictions from your model to your FiftyOne dataset\n",
    "-   Computing insights into your dataset relating to possible mistakes\n",
    "-   Visualizing the mistake in the FiftyOne App"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "This tutorial requires [PyTorch](https://pytorch.org) to be installed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify as necessary (e.g., GPU install). See https://pytorch.org for options\n",
    "!pip install torch\n",
    "!pip install torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll also need to download a pretrained CIFAR-10 PyTorch model (a ResNet-50) from the web:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'PyTorch_CIFAR10'...\n",
      "remote: Enumerating objects: 551, done.\u001b[K\n",
      "remote: Total 551 (delta 0), reused 0 (delta 0), pack-reused 551\u001b[K\n",
      "Receiving objects: 100% (551/551), 6.54 MiB | 3.20 MiB/s, done.\n",
      "Resolving deltas: 100% (182/182), done.\n",
      "Downloading '1dGfpeFK_QG0kV-U6QDHMX2EOGXPqaNzu' to 'PyTorch_CIFAR10/cifar10_models/state_dicts/resnet50.pt'\n",
      " 100% |████|  719.8Mb/719.8Mb [36.2s elapsed, 0s remaining, 24.4Mb/s]      \n"
     ]
    }
   ],
   "source": [
    "# Download the software\n",
    "!git clone https://github.com/huyvnphan/PyTorch_CIFAR10\n",
    "\n",
    "# Download the pretrained model (90MB)\n",
    "!eta gdrive download --public \\\n",
    "    1dGfpeFK_QG0kV-U6QDHMX2EOGXPqaNzu \\\n",
    "    PyTorch_CIFAR10/cifar10_models/state_dicts/resnet50.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manipulating the data\n",
    "\n",
    "For this walkthrough, we will artificially perturb an existing dataset with\n",
    "mistakes on the labels. Of course, in your normal workflow, you would not add\n",
    "labeling mistakes; this is only for the sake of the walkthrough.\n",
    "\n",
    "The code block below loads the test split of the CIFAR-10 dataset into FiftyOne\n",
    "and randomly breaks 10% (1000 samples) of the labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split 'test' already downloaded\n",
      "Loading 'cifar10' split 'test'\n",
      " 100% |█████████████████████████| 10000/10000 [2.3s elapsed, 0s remaining, 4.2K samples/s]      \n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "import fiftyone as fo\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "# Load the CIFAR-10 test split\n",
    "# Downloads the dataset from the web if necessary\n",
    "dataset = foz.load_zoo_dataset(\"cifar10\", split=\"test\")\n",
    "\n",
    "# Get the CIFAR-10 classes list\n",
    "info = foz.load_zoo_dataset_info(\"cifar10\")\n",
    "classes = info.classes\n",
    "\n",
    "# Artificially corrupt 10% of the labels\n",
    "_num_mistakes = int(0.1 * len(dataset))\n",
    "for sample in dataset.take(_num_mistakes):\n",
    "    mistake = random.randint(0, 9)\n",
    "    while classes[mistake] == sample.ground_truth.label:\n",
    "        mistake = random.randint(0, 9)\n",
    "\n",
    "    sample.tags.append(\"mistake\")\n",
    "    sample.ground_truth = fo.Classification(label=classes[mistake])\n",
    "    sample.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print some information about the dataset to verify the operation that we\n",
    "performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:           cifar10-test\n",
      "Persistent:     False\n",
      "Num samples:    10000\n",
      "Tags:           ['mistake', 'test']\n",
      "Sample fields:\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n"
     ]
    }
   ],
   "source": [
    "# Verify that the `mistake` tag is now in the dataset's schema\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 ground truth labels are now mistakes\n"
     ]
    }
   ],
   "source": [
    "# Count the number of samples with the `mistake` tag\n",
    "num_mistakes = len(dataset.match_tag(\"mistake\"))\n",
    "print(\"%d ground truth labels are now mistakes\" % num_mistakes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add predictions to the dataset\n",
    "\n",
    "Using an off-the-shelf model, let's now add predictions to the dataset, which\n",
    "are necessary for us to deduce some understanding of the possible label\n",
    "mistakes.\n",
    "\n",
    "The code block below adds model predictions to another randomly chosen 10%\n",
    "(1000 samples) of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torchvision\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "import fiftyone.utils.torch as fout\n",
    "\n",
    "sys.path.insert(1, \"PyTorch_CIFAR10\")\n",
    "from cifar10_models import *\n",
    "\n",
    "\n",
    "def make_cifar10_data_loader(image_paths, sample_ids, batch_size):\n",
    "    mean = [0.4914, 0.4822, 0.4465]\n",
    "    std = [0.2023, 0.1994, 0.2010]\n",
    "    transforms = torchvision.transforms.Compose(\n",
    "        [\n",
    "            torchvision.transforms.ToTensor(),\n",
    "            torchvision.transforms.Normalize(mean, std),\n",
    "        ]\n",
    "    )\n",
    "    dataset = fout.TorchImageDataset(\n",
    "        image_paths, sample_ids=sample_ids, transform=transforms\n",
    "    )\n",
    "    return DataLoader(dataset, batch_size=batch_size, num_workers=4)\n",
    "\n",
    "\n",
    "def predict(model, imgs):\n",
    "    logits = model(imgs).detach().cpu().numpy()\n",
    "    predictions = np.argmax(logits, axis=1)\n",
    "    odds = np.exp(logits)\n",
    "    confidences = np.max(odds, axis=1) / np.sum(odds, axis=1)\n",
    "    return predictions, confidences, logits\n",
    "\n",
    "\n",
    "#\n",
    "# Load a model\n",
    "#\n",
    "# Model performance numbers are available at:\n",
    "#   https://github.com/huyvnphan/PyTorch_CIFAR10\n",
    "#\n",
    "\n",
    "model = resnet50(pretrained=True)\n",
    "model_name = \"resnet50\"\n",
    "\n",
    "#\n",
    "# Extract a few images to process\n",
    "# (some of these will have been manipulated above)\n",
    "#\n",
    "\n",
    "num_samples = 1000\n",
    "batch_size = 20\n",
    "view = dataset.take(num_samples)\n",
    "image_paths, sample_ids = zip(\n",
    "    *[(s.filepath, s.id) for s in view.iter_samples()]\n",
    ")\n",
    "data_loader = make_cifar10_data_loader(image_paths, sample_ids, batch_size)\n",
    "\n",
    "#\n",
    "# Perform prediction and store results in dataset\n",
    "#\n",
    "\n",
    "for imgs, sample_ids in data_loader:\n",
    "    predictions, _, logits_ = predict(model, imgs)\n",
    "\n",
    "    # Add predictions to your FiftyOne dataset\n",
    "    for sample_id, prediction, logits in zip(sample_ids, predictions, logits_):\n",
    "        sample = dataset[sample_id]\n",
    "        sample.tags.append(\"processed\")\n",
    "        sample[model_name] = fo.Classification(\n",
    "            label=classes[prediction], logits=logits,\n",
    "        )\n",
    "        sample.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print some information about the predictions that were generated and how\n",
    "many of them correspond to samples whose ground truth labels were corrupted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added predictions to 1000 samples\n",
      "94 of these samples have label mistakes\n"
     ]
    }
   ],
   "source": [
    "# Count the number of samples with the `processed` tag\n",
    "num_processed = len(dataset.match_tag(\"processed\"))\n",
    "\n",
    "# Count the number of samples with both `processed` and `mistake` tags\n",
    "num_corrupted = len(dataset.match_tag(\"processed\").match_tag(\"mistake\"))\n",
    "\n",
    "print(\"Added predictions to %d samples\" % num_processed)\n",
    "print(\"%d of these samples have label mistakes\" % num_corrupted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find the mistakes\n",
    "\n",
    "Now we can run a method from FiftyOne that estimates the mistakenness of the\n",
    "ground samples for which we generated predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing mistakenness for 1000 samples...\n",
      " 100% |███████████████████████████| 1000/1000 [1.3s elapsed, 0s remaining, 808.1 samples/s]         \n",
      "Mistakenness computation complete\n"
     ]
    }
   ],
   "source": [
    "import fiftyone.brain as fob\n",
    "\n",
    "# Get samples for which we added predictions\n",
    "h_view = dataset.match_tag(\"processed\")\n",
    "\n",
    "# Compute mistakenness\n",
    "fob.compute_mistakenness(h_view, model_name, label_field=\"ground_truth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above method added `mistakenness` field to all samples for which we added\n",
    "predictions. We can easily sort by likelihood of mistakenness from code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:        cifar10-test\n",
      "Num samples:    1000\n",
      "Tags:           ['test', 'processed', 'mistake']\n",
      "Sample fields:\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n",
      "    resnet50:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)\n",
      "    mistakenness: fiftyone.core.fields.FloatField\n",
      "Pipeline stages:\n",
      "    1. <fiftyone.core.stages.MatchTag object at 0x7f9cb80dbc50>\n",
      "    2. <fiftyone.core.stages.SortBy object at 0x7f9d4bbd14e0>\n"
     ]
    }
   ],
   "source": [
    "# Sort by likelihood of mistake (most likely first)\n",
    "mistake_view = (dataset\n",
    "    .match_tag(\"processed\")\n",
    "    .sort_by(\"mistakenness\", reverse=True)\n",
    ")\n",
    "\n",
    "# Print some information about the view\n",
    "print(mistake_view)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Sample: {\n",
      "    'dataset_name': 'cifar10-test',\n",
      "    'id': '5ef384e36696dbdeabc6a88e',\n",
      "    'filepath': '/home/voxel51/fiftyone/cifar10/test/data/00107.jpg',\n",
      "    'tags': BaseList(['test', 'processed']),\n",
      "    'ground_truth': <Classification: {'label': 'deer'}>,\n",
      "    'resnet50': <Classification: {\n",
      "        'label': 'horse',\n",
      "        'logits': array([-0.83586901, -1.28598607,  1.54965878, -0.49650264, -0.40103185,\n",
      "               -0.18043809, -1.0332154 ,  5.05314684, -1.21831954, -1.15143788]),\n",
      "    }>,\n",
      "    'mistakenness': 1.0,\n",
      "}>\n",
      "<Sample: {\n",
      "    'dataset_name': 'cifar10-test',\n",
      "    'id': '5ef384e36696dbdeabc6a86f',\n",
      "    'filepath': '/home/voxel51/fiftyone/cifar10/test/data/00076.jpg',\n",
      "    'tags': BaseList(['test', 'processed']),\n",
      "    'ground_truth': <Classification: {'label': 'bird'}>,\n",
      "    'resnet50': <Classification: {\n",
      "        'label': 'deer',\n",
      "        'logits': array([-0.72157425, -0.94043797, -0.32308894, -0.19049911,  4.82478857,\n",
      "               -0.35608411, -0.35027471, -0.25426134, -0.77823019, -0.91033494]),\n",
      "    }>,\n",
      "    'mistakenness': 1.0,\n",
      "}>\n",
      "<Sample: {\n",
      "    'dataset_name': 'cifar10-test',\n",
      "    'id': '5ef384e36696dbdeabc6a838',\n",
      "    'filepath': '/home/voxel51/fiftyone/cifar10/test/data/00021.jpg',\n",
      "    'tags': BaseList(['test', 'mistake', 'processed']),\n",
      "    'ground_truth': <Classification: {'label': 'frog'}>,\n",
      "    'resnet50': <Classification: {\n",
      "        'label': 'deer',\n",
      "        'logits': array([-0.77428126, -1.11018133,  1.21526551, -0.23978873,  3.74053574,\n",
      "               -0.37081209,  0.20087151, -0.54353052, -1.05138922, -1.06668639]),\n",
      "    }>,\n",
      "    'mistakenness': 1.0,\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Inspect the first few samples\n",
    "print(mistake_view.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the App to visually inspect the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "App launched\n"
     ]
    }
   ],
   "source": [
    "# Launch the FiftyOne App\n",
    "session = fo.launch_app()\n",
    "\n",
    "# Open your dataset in the App\n",
    "session.dataset = dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![dataset](images/label_mistakes_1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show only the samples that were processed\n",
    "session.view = dataset.match_tag(\"processed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![processed](images/label_mistakes_2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show only the samples for which we added label mistakes\n",
    "session.view = dataset.match_tag(\"mistake\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![mistake](images/label_mistakes_3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the samples we processed in rank order by the mistakenness\n",
    "session.view = mistake_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![mistake-view](images/label_mistakes_4.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
